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TOWARD THE DEVELOPMENT OF A COMPREHENSIVE EVALUATION DESIGN 
Preliminary Draft of Presentation for AERA Special Interest Group in 

Teacher Education 

Teacher Education Centers: University of Maryland 
James Greenberg 
Ruth Heidelbach 
Judith Ruchkin 

Introduction 

In 1968, the University of Maryland received the AACTE Award for 
the Teacher Education Center concept, one dimension of the professional 
component of a total program preparing teachers in all areas of elemen- 
tary and secondary education. Beginning with a limited number of sites 
in one Maryland counly, this effort has grown to encompass fifteen 
centers in seven counties and municipalities in Maryland and the District 
of Columbia within six years. 

t 

We, who are currently charged with the most direct responsi- 
bility for the University of Maryland's part of the Center effort, 
are new to the scene. Our relation to the operation might be character- 
'ized as new leadership, a function with the implicit advantages and 
limitations that accrue to such a role. Consequently, this submission- 
will attempt to define purposes, strategies, and the history of evalu- 
ation of the Center effort as we have come to know it and will present 
our current thinking about evaluation from this particular perspective. 

Description of the Center Effort 

What can be said about the Center effort derives from the literature 
and reports which reside at the University and from the perceptions of 
those persons who played and continue to play some part in the creation 
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and maintenance of the Teacher Education Centers. Public School systems 
and the University have joined together in mutual selection of center 
schools, center coordinators, and definition of project parameters and 
rold functions. 

Students obtain pre-service experience through cooperation of a 
variety of Center personnel instead of being assigned to one supervising 
teacher. Hiis arrangement provides a potential for flexibility and individ- 
uality in each student's experience and is specifically manifested in 
intensive and extensive experiences on multiple levels in Center schools. 
Clinical supervision is shifted toward the coordinator and school personnel 
in a form different from traditional school and college supervision. 
According to the plan, coordinators and college supervisors function 
as trainers of teacher trainers. Seminars, skill development, and extensive 
feedback are other components of pre-service strategies in Centers. An 
array of in-service offerings, professional materials, equipment, and 
travel replaces the usual monetary stipend for Center supervising teachers. 
Mutual benefit is expected from these arrangements. 

Our brief involvement with The Maryland Teacher Education Center 
effort begets admiration for the extent of innovative movement involved 
in the development and growth of the Centers. Institutions are hard to 
change, and the commitments of this large University and the related public 
school systems in this venture reflect an achievement that is notable in and 
of itself. 

From the two sponsoring institutional sources and from direct contact 
and observation, a summary of purposes and evaluation can be cited. 
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PURPOSES AND STRATEGIES 



Tlie original purposes can be classified in three categories. 

Process , The center effort is intended to: unify pre- and in- 
service teacher preparation into continuous study for individuals in 
training; bridge the gap between University and school systems and 
provide a means for articulation and mutual involvement; utilize technology 
and knowledge of learning systems in a more effective professional component 
provide a locus for research and knowledge production. 

Organization . Ihe organizational structure is intended to estab- 
lish cooperative units between the University and public school systems, 
which are jointly funded and operated; to establish the position of a 
Teacher Education Center coordinator, jointly employed by the two systems, 
who acts as liaison and administrative leader of his individual center; 
and to join two or more schools into a center linked through a conceptual 
base . 

Outcomes . 1 1 is intended that the outcomes will be: teachers 

who can demonstrate flexibility, individual teaching styles, and a 
variety of teaching strategies; practitioners who continually renew 
and improve their teaching; and proficient clinical trainers. 

EVALUATION 

To our knowledge, evidence does not exist that a comprehensive 
evaluation design was built into the original Center effort at the 
time of its initiation. Rather, energies have been devoted to the 
important aspects of growth and expansion as well as program develop- 
ment. Over the years, attention has been given to evaluation. School 
and college personnel have reported investigations and findings. One 



form of “evaluation" has been the ongoing sharing and reporting of 
developments by coordinators and organized planning groups which 
provides a minimal feedback mechanism for decision makers. Some atten- 
tion was given to systematic evaluation through the appointment of a 
research coordinator, (Currently, this position does not exist.) 

Findin^is from work done at that time, reported at the 1970 AERA Conference, 
indicated that "student teachers in Centers do teach differently and 
hold different attitudes than their non-Center peers 

Evaluiation Design ; Emerging Frameworks 

While new leaders who inherit projects do not have the luxury of 
designing an evaluation component as an integral part of an original 
plan, some design possibilities can be anticipated based on the needs 
as perceived in the current context. Decisions, innovations and modifi- 
cations continue to be required. 

Instead of drawing upon outside evaluators with particular skills, 
interests, and role definitions, which places its own kind of constraint 
upon evaluation, emerging evaluation frameworks are derived here directly 
from the vantage point of current decision-makers. It should be made 
explicit that certain of the participating decision makers are most recently 
project inheritors and are outside heirs of the Center effort. Thus, 
the requirements for gathering information, i.e. knowledge calculated 
to reduce uncertainty in decision making, in contrast to collecting 
data unrelated to administrative values and goals ,are considerable . 

^David B. Young. "Teacher Education Centers: Do They Make a Difference. 
Paper read at AERA Annual Meeting, Chicago, 1970. 
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It also seems useful, for our purposes at this time, to distin- 
quish between evaluation and research. Essentially, the former has as 
its primary focus the determination of either absolute or comparative 
worth, while the latter is concerned with the production of new 
knowledge. In this framework then, evaluation may, but does not neces- 
sarily, contribute to the advancement of knowledge.^ Its primary criterion 
is informational utility for decision-making. Thus, the possible dis- 
covery of insignificant relationships between the performance produced 
by the standard and the innovative program would constitue valuable data, 
i.e. information, for decision-making with particular reference to rela- 
tive costs reflecting program commitments. However, such findings would 
not have knowledge implications beyond the immediate environment. 

Mindful of the need for systematic inquiry into teaching and its poten- 
tial utility to practitioners as well as researchers and even desirous 
of contributing to that process, it nonetheless appears necessary to 
distinquish between the two related but not synonymous tasks of research 
and evaluation. As a practical matter, merging the separate function 
of evaluation with attempts at creating knowledge may result in doing 
neither task adequately. 



both the rationale and the second evaluation framework in this 
paper owe a considerable debt to: Michael Scriven '*The Methodology of 

Evaluation'’ in Ralph Tyler, et. al. Perspectives of Curriculum Evaluation , 
AERA monograph Series on Curriculum Evaluation No. 1, Chicago, Rand McNally, 
1967, pp. 39-83 and Robert Stake "Tlie Countenance of Educational Eval- 
uation”, Teachers College Record , 68 (April, 1967), pp. 523-540. 




6 



6 



Basic Questions 

The context into which evaluation now fits involves decision 
makers in several public school systems and at the University of 
Maryland. To be worthwhile, an evaluation design will have to answer 
some basic questions for these decision makers and will have to gen- 
erate studies and information directly related to effective decision 
making . 

Basic questions that need to be answered are as follows: 

Was the original Teacher Education Center plan implemented 

and to what degree? 

Does the project affect : 

1. measurable gains in specified teacher behaviors? 

2. measurable gains in specified pupil outcomes? 

3. direct or indirect changes in professional sequence? 

Should the project, or any of its constituents or component parts 

1. continue as it is? 

2. be replicated? 

3. be revised somewhat or extensively? 

4. be terminated? 




Emerging Conceptual Frameworks 

At least two emerging conceptual frameworks seem to reflect our 
thinking to date about useful evaluation design possibilities. In light 
of the planning-decision questions to be answered, a design will have 
to reveal not only where the Teacher Education Center effort is in terms 
of its original and present conception, but also whether or not its 

. . . ^ 
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foc\!S and goals are In keeping v;ith current thinking about teacher 
education. 1 1 is hoped that any evaluation process will for\%^ard the 
development of emergent conceptions of future instructional roles and assist 
in casting teacher preparation efforts squarely in the context of studies 
of teaching behavior. 

hoc evaluation effort is weakened by all the well 
known drawbacks for entering a field lacking specific provisions for 
evaluation concurrent with project development. However, an under- 
standing of the project’s past appears to have utility for current 
decision-making. Conscious of the need for gathering information 
bearing upon present decisions it also appears logically possible 
to use current status assessment as baseline measures for any new program 
versions or variations that may develop. 

Several criteria applicable to the frameworks for evaluation 
designs discussed here need to be specified: 

1. They are open ended, i.e. make provision for inclusion of new 
knowledge, values, individuals. . . 

2. They have future focus and feedback potential for a wide 
range of decision-makers . 

3. They are timely in anticipating future issues requiring 
prompt decisions. 

4. They search for both goal achievement and failure. 

5. They monitor for unintended consequences. 

6. They emphasize controllable components. 

Evaluation designs should provide evidence on central questions 
such as: (1) Are there particularly successful as well as unsuccessful 
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Centers (2) What components are responsible for relative success or 
failure? (3) Are there identifiable groups, subpopulations, exhibiting 
substantial gains or losses? 

A first framework possibility would be a comprehensive matrix (see 
Figure 1) developed for the purpose of generating evaluation questions 
and studies along several criteria levels. On the vertical axis, stated 
and implied project goals represent the specific foci for study and 
evaluation. On the horizontal axis, levels of criteria are designated 
around which different degrees of design elements can be developed. 

Turner's six criteria levels^ are included in descending hierarchical 
order. A seventh level is added for the purpose of generating historical 
and status studies for needed background data. This level of investiga- 
tion should develop documentation information about past and present 
implementation levels of stated intentions as well as provide subjective 
attitudinal data from affected parties. 

The matrix generates cells, each of which ties a specific purpose 
with a specific level of evaluation. Two categories of results of this 
pairing will be possible. First, each purpose will be scrutinized for 
evaluation potential at each level. Performance criteria require per- 
formance-based objectives and so determinations about the quality of each 
purpose will be forthcoming in these terms. Where purposes, do not 
lend themselves to evaluation in performance terms, background redefi- 
nition tasks will be indicated, along with determinations about 
needed data on objectives, for evaluation to proceed at a fruitful 

^Excerpt from Power of Competency Based Teacher Education . Final 
Report of the Committee on National Program Priorities in Teacher Education 
(CNPPTE) Benjamin Rosner,- Chairman. United States Office of Education, 1971. 
(See Appendix) 

ST 
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Comprehensive Matrix for Generating 
Evaluation Questions and Studies 
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level. Second, questions and studies will emerge in each cell as a basis 
for individual designs. Following this, steps can be included in each 
cell outlining appropriate indicators, format and instruments for data 
gathering, feasibility of data collection on time, cost and personnel 
dimensions, analytical steps, and procedures for feeding results into 
planning decision contexts. For instance, an obviously limited example 
of foci which could be generated in these cells is outlined in Figure 2. 

Current systematic evaluation studies are focused on (1) simulation 
strategies in relation to selected supervisor and student teacher be- 
havior, and (2) analytical studies of supervisory conference behaviors 
and the effects of self-determined feedback upon classifications of 
supervisory interaction developed by Heidelbach.^ These studies might 
be classified in the Comprehensive Matrix Cells A5 and C4 respectively. 
(See Figure 1) Center Coordinators are collaborating with college 
personnel in carrying out these investigations. 

A second conceptual framework utilizes implementation or consis- 
tency analysis. This offers another possibility for evaluation that 
meets the previously stated criteria. In the context of joint school- 
university operation of the Teacher Education Centers, which are the sub- 
ject of the evaluation, consistency analysis would encompass multiple 
levels and, at least, dual institutional perspectives. 
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Ruth Heidelbach, "The Cooperating Teacher as Teaching Tutor," 
Inquiry into Teaching Behavior of Supervisors in Teacher Education 
Laboratories . Margaret Lindsey and Associates, (New York; Teachers 
College Press, 1969). 



11 
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The Strengthening of In-Service Teacher Education Programs 
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It would match (1) goals or plan, (2) instructional strategies 
or program cind (3) tested-for or expected performance. Within the 
performance category it would be important to assess match bem^een 
tested-for and expected performance by both institutional sponsors. 

A partial list of appropriate items for the three categories 
is found below: j 







GOAL 


PROGRAM 


PERFORl-lANCE 

4 - - - - 




1. 


Unify Pre- and In- 
Service preparation 


Coordinator assessment 

of student & staff , 
needs 


Continuous professional 
growth 




2. 


Integrate theory 
with practice 


Relate on and off*- 
camp us segments th ro ugl 
coordinator 


i 




3. 


Individualized pro- 
fessional deve^opm- 


Intensive and exten- 
sive experiences ; in- 
service offerings 


Flexibility, individual 
style 




4. 


Systematic analysis 
of classroom be- 
haviors 


Professional Seminars, 
materials , equipment 
and travel 


Variety of teaching 
strategies 




5. 


Utilize educational 
technology 


Microteaching 


Proficient clinical 
trainers and practicioners 
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Lqcus for: research 
and model programs 







FIGURE 3 

Consistency Analysis 
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A four phase investigation conducted by independent workers 
engaging in parallel inquiry is proposed. The first phase would review 
logical consistency and/or conflict among goals. This explores congruence 
and/or divergence within and among (a) explicit, (b) implicit and 
(c) tested-for goals. The second, third and fourth phases, going in 
tandem with this necessary goal review are examinations of match between; 
goals and program; goals and tested-for or expected performance; and 
program and tested-for or expected performance. A graphic representation 
of these three match models is found in Figure 4. It should be noted that 
if there is congruency among goals and program and goals and performance that 
consistency between program and performance would logically follow. However, 
phase four is included as an internal empirical check on the previous phases. 

For purposes of illustration it may be useful to consider the match 
among a few items representing past goals, presented in Figure 5^ Explicit 
goals are derived from documentary statements. Implicit goals are 
retrospective assertions about program objectives and tested- for goals are 
measures regularly employed in monitoring the program components. The 
analysis would ascertain the logical connection' among the three goal dimensions 
cited. This analysis needs to include dual institutional perspectives and 
it may also focus on developments over time. While the previous examples 
are merely intended for purposes of illustration it is clear that the original 
center plan did not in all cases provide for tested-for goals to accompany 
its explicit or stated objectives. 
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Graphic Representation of Match Models 
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explicit goal 


implicit goal 


tested-for goal 


1. Establish jointly 
funded and oper- 
ated centers 


bring school and 
university closer 
together 


in-service enroll- 
ment 


2. create coordinator 
liason 


- reduce college 
faculty super- 
vising load 


pre-service partici- 
pation of school 
personnel 


3. provide variety 
of student 
placements 


wider exposure to 
practices 
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Figure 5 

Consistency Analysis 


of Goals 



The outcome of following such an evaluation design is data- 
grounded decision-making. If goals are still deemed worthy; and if 
there is match among goals, program and performance; and if influence 
of new knowledge, values and individuals is also consistent with the 
afore-mentioned process then adequate information has been provided for 
(a) continuance and (b) replication. However if match is lacking 
and there are no intervening influences, then information has been provided 
for any one or combination of actions such as: (a) revise goal (b) revise 

program components to fit goal, (c) start anew, or (d) ignore discrepancy. 

Prior to the adoption of any one or combining conceptual frameworks there 
needs to be review of the projected effort. It is also hoped that the SIG 
Teacher Education session will provide a significant opportunity for feedback 
and interchange and for rendering different evaluation studies and designs 
useful to workers in other contexts. A means for establishing comparability 
of evaluation designs and findings would appear essential so that studies on 
limited localized populations and programs may be pooled and thereby gain 
greater generalizability for decisiorimaking , program development and practice. 
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